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(57) Abstract 

There is disclosed a surveillance and monitoring 
system and method for monitoring an area. The system 
includes a first imaging system having a wide-angle field 
of view approximately equal to or greater than the area. 
The system also includes one or more second imaging 
systems having adjustable view settings, each of said one or 
more second imaging systems positioned to view portions 
of said area and being capable of producing images of said 
portions with a resolution greater than said first imaging 
system. The system also includes one or more controls for 
controlling the adjustable view settings of the ^ne or more 
second imaging systems, so as to obtain high- resolution, 
magnified images of regions of interest within the area being 
monitored. In a preferred embodiment, the adjustable view 
settings include pan, tilt, and zoom settings for adjusting 
the panning, tilting, and zooming of the one or more second 
imaging systems. 
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A COMBINED WIDE ANGLE AND NARROW ANGLE IMAGING SYSTEM 
AND METHOD FOR SURVEILLANCE AND MONITORING 



SPECIFICATION 



NOTICE OF GOVERNMENT RIGHTS 
5 The U.S. Government has certain rights in this invention pursuant to the terms of 

DARPA Contract No. N00014-97-1-0553 and National Science Foundation Grant No. 
IRI-93-57594. 

MICROFICHE APPENDIX 
A microfiche appendix of a computer program for performing the techniques in 
10 accordance with the invention is attached to this document. There is a total of one 
microfiche having 98 frames as part of the appendix. 



BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

This invention relates to a system and method for monitoring and surveillance, 
15 and particularly to a system and method employing both wide-angle and narrow-angle 
imaging optics. 

2. Discussion of the State of the Art 

Traditionally, surveillance and monitoring ("SAM") systems have used off-theT 
shelf lenses and cameras for imaging. Because of the limitations of lenses, these systems 
20 typically provide very limited fields of view. To increase their limited fields of view, 
traditional SAM systems have relied on panning and tilting of the imaging system. As 
defined in this specification and the appended claims, "panning" refers to movement in a 
plane substantially horizontal to the area being monitored, and "tilting" refers to 
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movement in a plane substantially vertical to the area being monitored. Typically, when 
a pan-and-tilt system is used, the system scans an area in some predefined or random path 
until an object of interest is detected. At that point, the object may be tracked by the 
system for further observation. Typically, such a system may also include a zoom lens 

5 for zooming in on objects of interest. Such systems are usually characterized as pan, tilt, 
and zoom ("PTZ") systems. Another approach to SAM systems has been the use of 
multiple PTZ systems to cover an area being monitored. Examples of PTZ systems are 
described in U.S. Patent No. 5,627,616 to Sargeant et al.; U.S. Patent No. 5,394,209 to 
Stiepel et al.; U.S. Patent No. 5,164,827 to Paff; and U.S. Reissue Patent No. 34,989 to 

1 0 Struhs et al., which are incorporated herein by reference. 

While PTZ systems enlarge the field of view capable of being monitored by a 
lens-and-camera system, their scanning time makes them unsuitable for many real-time 
applications— i.e., fast-moving objects may enter and leave the area being monitored 
before the PTZ systems are able to detect them. Additionally, if a PTZ system uses a 

15 predefined scanning path, the monitoring performed by the system could be 

circumvented. That is, if an intruder is aware of the predefined scanning path, the 
intruder may be able to move about the monitored area without being detected by the 
PTZ system. 

Another approach to SAM systems has been the use of wide-angle or so-called 
20 "omnidirectional" imaging systems. For example, the use of a "fish-eye" lens for wide- 
angle viewing is disclosed in E.L. Hall et al., "Omnidirectional Viewing Using a Fish Eye 
Lens", SPIE Vol. 728 Optics, Illumination, and Image Sensing for Machine Vision 
(1986), p. 250, and U.S. Patent No. 5,185,667 to Zimmerman, which is incorporated 
herein by reference. Since the fish-eye lens has a very short focal length, the field of 
25 view may be as large as or sometimes greater than a hemisphere. 

Other prior art devices have used reflecting surfaces to increase the field of view. 
One such prior art device is disclosed in V.S. Nalwa, "A True Omni-Directional Viewer", 
AT&T Bell Laboratories Technical Memorandum, BL01 15500-9601 15-01, Jan. 1996. 
Nalwa discloses the use of multiple planar reflecting surfaces in conjunction with 
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multiple charge-coupled device ("CCD") cameras to obtain a 360 degree panoramic 
image of a 50 degree band of a hemispherical scene. Specifically, in Nalwa, four planar 
mirrors are arranged in the shape of a pyramid, with one camera being positioned above 
each of the four planar reflecting sides, and with each camera viewing slightly more than 

5 90 degrees by 50 degrees of the hemispherical scene. A similar device is disclosed in 
U.S. Patent No. 5,539,483 to Nalwa, which is incorporated herein by reference. 

Both Yagi et al., "Evaluating Effectivity of Map Generation by Tracking Vertical 
Edges in Omnidirectional Image Sequence", IEEE International Conference on Robotics 
and Automation, June 1995, p. 2334, and Yagi et al., "Map-Based Navigation for a 

10 Mobile Robot With Omnidirectional Image Sensor COPIS", IEEE Transactions on 

Robotics and Automation, Vol. II, No. 5, Oct. 1995, disclose a conical projection image 
sensor (COPIS) which uses a conical reflecting surface to gather images from the 
surrounding environment and processes the information to guide the navigation of a 
mobile robot. 

15 Yamazawa et al., "Obstacle Detection With Omnidirectional Image Sensor 

HyperOmni Vision", IEEE International Conference on Robotics and Automation, Oct. 
1995, p. 1062, discloses a purported improvement to the COPIS system which involves 
the use of a hyperboloidal reflecting surface in place of a conical surface. Prior to 
Yamazawa et al., U.S. Patent No. 3,505,465 to Donald Rees also disclosed the use of a 

20 hyperboloidal reflecting surface to achieve panoramic viewing. Rees is incorporated 
herein by reference. 

As compared to traditional PTZ systems, the wide-angle or omnidirectional prior 
art devices described above have certain disadvantages. For example, the wide-angle or 
omnidirectional devices typically provide image resolution that is relatively low as 
25 compared to traditional PTZ systems. This is because, to avoid costly special designs, the 
wide-angle or omnidirectional devices typically utilize off-the-shelf cameras whose 
resolution is adequate for smaller fields of view. In addition, a lower resolution is often 
necessary if real-time video images are desired from the wide-angle or omnidirectional 
devices because real-time, high resolution video images of a wide field of view require a 
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great amount of throughput on the part of image processing equipment connected to the 
wide-angle or omnidirectional devices. Moreover, a further drawback of wide-angle or 
omnidirectional devices as compared to traditional PTZ systems is that zooming in on a 
region of interest by image processing of a wide-angle or omnidirectional image cannot 
5 provide better resolution of the region of interest than in the original wide-angle or 

omnidirectional image, whereas zooming in on a region of interest with a zoom lens of a 
traditional PTZ system can provide higher resolution of the region of interest than in the 
original image. 

Accordingly, there exists a need for a cost-effective SAM system that provides 
10 both a wide-angle field of view of an area being monitored in combination with the 
capability for high-resolution images of regions of interest within the monitored area. 

SUMMARY OF THE INVENTION 
The primary objective of the present invention is to provide a surveillance and 
monitoring system and method that provides both a wide-angle field of view of an area 
15 being monitored as well as the capability for high-resolution, magnified images of 

regions of interest within the monitored area. Other objectives will become apparent to 
those skilled in the art as they read the specification herein. 

To overcome the drawbacks of the prior art, a surveillance and monitoring system 
according to the present invention includes a first imaging system having a wide-angle 
20 field of view approximately equal to or greater than the area being monitored. The 
system also includes one or more second imaging systems having adjustable view 
settings, each of the second imaging systems positioned to view portions of the area and 
being capable of producing images of the portions with a greater resolution than the first 
imaging system. The system also includes one or more controls for controlling the 
25 adjustable view settings of the one or more second imaging systems. 

In use, the first imaging system provides a wide-angle view of the area being 
monitored. The wide angle view is then used to control the adjustable view settings of 
the second imaging systems, which are capable of providing greater resolution images of 
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regions of interest within the area. 

In a preferred embodiment of the surveillance and monitoring system of the 
present invention, the adjustable view settings include pan, tilt, and zoom settings for 
adjusting the panning, tilting, and zooming of the one or more second imaging systems 
5 with respect to the area being monitored. 

In another preferred embodiment of the surveillance and monitoring system of the 
present invention, the area is substantially hemispherical and the first imaging system has 
a substantially hemispherical field of view. 

In another preferred embodiment of the surveillance and monitoring system of the 
1 0 present invention, the first imaging system is a catadioptric system, which includes a 
reflecting surface positioned to reflect an image of the area being monitored and one or 
more image sensors optically coupled to the reflecting surface. Most preferably, the 
reflecting surface of the catadioptric system is a truncated, substantially paraboloid- 
shaped mirror. Alternatively, the reflecting surface of the catadioptric system may 
15 include a substantially hyperboloidal-shaped mirror, a substantially ellipsoidal-shaped 
mirror, one or more curved mirrors, or one or more planar mirrors. 

Instead of catadioptric systems, the first imaging system may include wide-angle 
imaging optics coupled to one or more image sensors. Such optics may include a wide- 
angle lens or a fish-eye lens. In addition, the first imaging system may include a plurality 
20 of camera systems. 

In yet another preferred embodiment of the present invention, a monitoring station 
is provided, which includes one or more displays, at least one display coupled to the first 
imaging system for displaying an image of the area being monitored and at least one 
display coupled to the one or more second imaging systems for displaying a region of 
25 interest within the area. The monitoring station includes input means for selecting the 
region of interest from the display coupled to the first imaging system. The input means 
is operatively coupled to the controls for controlling the adjustable view settings, such 
that at least one of the second imaging systems is positioned to view the region of 
interest. 
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In yet another preferred embodiment of the present invention, the first imaging 
system provides an image signal representative of the area being monitored, and the 
surveillance and monitoring system further includes an image signal processing apparatus 
coupled to the first imaging system to convert the image signal into image signal data. 
5 Preferably, the image signal processing apparatus includes means for mapping the image 
signal data into a Cartesian-coordinate system to produce a perspective image or into a 
cylindrical-coordinate system to produce a panoramic image. 

In yet another preferred embodiment, the surveillance and monitoring system of 
the present invention further includes motion detection means coupled to the first 
10 imaging system for detecting objects in motion in the area being monitored. Preferably, 
the motion detection means is coupled to a tracking means for tracking one or more of the 
objects in motion. The tracking means may then be coupled to the controls for the 
adjustable view settings of the second imaging systems to view the objects being tracked 
with the second imaging systems. 
15 In yet another preferred embodiment of the present invention, a surveillance and 

monitoring system for monitoring an area is provided comprising: a wide-angle imaging 
system having a wide-angle field of view approximately equal to or greater than the area; 
motion detection means coupled to the imaging system for detecting objects in motion in 
the area; tracking means coupled to the motion detection means for tracking one or more 
20 of the detected objects in motion; and image processing means coupled to the tracking 
means and the wide-angle imaging system for producing perspective images of the 
tracked objects from images provided by the wide-angle imaging system. 

In accordance with the present invention, a method is also provided for 
monitoring an area. The method includes the steps of: sensing a global image of the area 
25 being monitored with a first imaging system having a field of view approximately equal 
to or greater than the area; detecting a region of in terest within the global image; and 
sensing one or more detailed images of the region of interest with at least one of the one 
or more second imaging systems, at least one of the detailed images having a higher 
resolution than the global image. 
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In another preferred embodiment of the present invention, a method is provided 
which includes the steps of: positioning a first imaging system to view the area, the first 
imaging system having a wide-angle field of view approximately equal to or greater than 
the area; positioning one or more second imaging systems having adjustable view settings 
5 to view portions of the area, each of the one or more second imaging systems being 
capable of producing images of the portions with a resolution greater than the first 
imaging system; sensing an image of the area with the first imaging system; detecting a 
region of interest within the sensed image; and controlling the adjustable view settings so 
as to view the region of interest with at least one of the one or more second imaging 
10 systems. 

In accordance with the present invention, a method for monitoring one or more 
objects in motion in an area is also provided, which includes the steps of: positioning a 
first imaging system to view the area, the first imaging system having a wide-angle field 
of view approximately equal to or greater than the area; positioning one or more second 

15 imaging systems having adjustable view settings to view portions of the area, each of the 
one or more second imaging systems being capable of producing images of the portions 
with a resolution greater than the first imaging system; sensing images of the area with 
the first imaging system; detecting the one or more objects in motion from the sensed 
images; tracking one or more of the detected objects; and controlling the adjustable view 

20 settings so as to view the tracked objects with at least one of the one or more second 
imaging systems. 

Preferably, the step of detecting in the method for monitoring one or more objects 
in motion includes the sub-steps of: generating a series of image frames at predetermined 
time intervals from the sensed images; filtering the series of image frames for noise; 
25 calculating a moving average frame from a subset of the series of image frames 

comprising the set from the first image frame to the next-to-last image frame; subtracting 
the filtered, last image frame of the series of image frames from the moving average 
frame to produce a difference image frame; comparing each pixel of the difference image 
frame to a predetermined threshold value to produce a threshold image frame indicative 
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of regions of motion in the area; defining a first group of objects associated with the 
subset of the series of image frames; associating a second group of objects with the 
regions of motion in the threshold image frame; and generating a third group of objects 
comprising all unique objects in the first and second groups. 

In a preferred embodiment, the tracking step in the method for monitoring one or 
more objects in motion includes the sub-steps of: selecting a fourth group of objects to be 
tracked from the third group of objects on the basis of a predetermined criteria; 
determining a focus point for each object in the fourth group of objects; and applying a 
smoothing function to the focus point of each object. 

Further, a preferred embodiment of the controlling step in the method for 
monitoring one or more objects in motion includes mapping the coordinates of the objects 
being tracked from the coordinate system of the first imaging system into the coordinate 
system of at least one of the second imaging systems. 

BRIEF DESCRIPTION OF THE DRAWINGS 
1 5 Exemplary embodiments of the present invention will now be described in detail 

with reference in the accompanying drawings in which: 

Fig. 1 is a side view of a video surveillance and monitoring system according to a 
preferred embodiment of the present invention; 

Fig. 2 is a functional block diagram of a video surveillance and monitoring system 
20 according to another preferred embodiment of the present invention; 
Fig. 3 A provides an illustration of perspective mapping; 
Fig. 3B provides an illustration of cylindrical mapping; 

Fig. 4 is a side view of a video surveillance and monitoring system according to 
another preferred embodiment of the present invention, in which the pan, tilt, and zoom 
25 camera is positioned directly underneath the wide-angle imaging system; 

Figs. 5A and 5B illustrate the mapping of object coordinates from the coordinate 
system of a wide-angle imaging system to the coordinate system of a pan, tilt, and zoom 
imaging system; 



5 



10 
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Fig. 6 is a functional block diagram of a video surveillance and monitoring system 
according to another preferred embodiment of the present invention; 

Figs. 7 A to 7C provide a flowchart of a preferred embodiment of a method for 
monitoring one or more objects in motion in an area; 
5 Fig. 8 is a side view of a video surveillance and monitoring system according to 

another preferred embodiment of the present invention, in which the wide-angle imaging 
system includes a catadioptric system with a paraboloid-shaped mirror; 

Fig. 9 is a side view of a video surveillance and monitoring system according to 
another preferred embodiment of the present invention, in which the wide-angle imaging 
1 0 system includes a catadioptric system with a hyperboloidal-shaped mirror; 

Fig. 10A is a side view of a video surveillance and monitoring system according 
to another preferred embodiment of the present invention, in which the wide-angle 
imaging system includes a catadioptric system with a plurality of planar mirrors arranged 
in a pyramid shape; 

15 Fig. 1 OB is a side view of a video surveillance and monitoring system according 

to another preferred embodiment of the present invention, in which the wide-angle 

imaging system includes a catadioptric system with a plurality of planar mirrors arranged 

in a polyhedral-shape; 

Fig. 1 1 is a side view of a video surveillance and monitoring system according to 
20 another preferred embodiment of the present invention, in which the wide-angle imaging 

system includes a wide-angle lens; 

Fig. 12 is a side view of a video surveillance and monitoring system according to 

another preferred embodiment of the present invention, in which the wide-angle imaging 

system includes a fish-eye lens; 
25 Fig. 1 3 is a perspective view of a video surveillance and monitoring system 

according to another preferred embodiment of the present invention, in which the pan, 

tilt, and zoom camera is mounted on a movable base; 

Fig. 14 is a side view of a video surveillance and monitoring system according to 

another preferred embodiment of the present invention, in which the pan, tilt, and zoom 
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camera contains a fixed camera and fixed optics and a movable mirror positioned 
between the camera and optics and the area to be monitored; and 

Figs. 15A and 15B are top plan views of video surveillance and monitoring 
systems according to preferred embodiments of the present invention, in which the wide- 
5 angle imaging system includes a plurality of camera systems. 

DETAILED DESCRIPTION 
Fig. 1 shows a preferred embodiment of the present invention. Arranged along a 
ceiling 5 is a wide-angle imaging system ("WAIS") 10, which has a field of view 
approximately equal to or greater than the area being monitored. Also arranged along the 

10 ceiling 5 are one or more pan, tilt, and zoom ("PTZ") imaging systems 20. Fig. 1 shows, 
for example, two such imaging systems arranged along the ceiling 5. Each PTZ system 
20 has a field of view smaller than that of the WAIS 10, but has a resolution that is 
relatively greater. As illustrated in Fig. 1, each PTZ system 20 includes optics 22, a 
camera 24 for sensing the images provided by the optics 22, a pivot arm 26 for movement 

15 of the camera 24 and optics 22, and a PTZ controller 27 mounted to the ceiling 5 for 
controlling the movement of the pivot arm 26. The pivot arm 26 is divided into two 
segments, a lower segment 26a connected to the camera 24 and an upper segment 26b 
pivotably mounted to the PTZ controller 27. The two segments 26a and 26b are joined 
by a pivot 28. In this embodiment, the pivot arm 26 provides each PTZ system 20 with a 

20 two-degree freedom of movement. Specifically, the pivotal connection of the upper 

segment 26b with the PTZ controller 27 provides each PTZ system 20 with the ability to 
pan around an axis perpendicular to the ceiling 5, and the pivot 28 between the and lower 
and upper segments 26a and 26b provides each PTZ system 20 with the ability to tilt with 
respect to the plane of the ceiling 5, Preferably, the optics 22 includes a zoom lens, 

25 which allows magnification of a region of interest within a PTZ system's field of view. 

In use, the WAIS 10 of the present invention provides a global view of the area 
being monitored. When a region of interest within the monitored area is detected in the 
global view provided by the WAIS 10, one or more of the PTZ systems 20 are moved to 
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view the region of interest and are used to obtain high-resolution, magnified images of 
that region. 

Although each PTZ system 20 of the preferred embodiment just described 
includes a pivot arm 26 providing a two-degree freedom of movement for each system, 
5 the presently claimed invention is, of course, not limited to such an embodiment. For 
example, a third degree of freedom could also be added to each PTZ system 20 by adding 
a means for rolling or rotating the camera 24 and optics 22 around the optical axis. In 
addition, the camera may have focus and iris settings. As shown in Fig. 13, a PTZ system 
20 could also include a movable base 29, which provides translational movement in three 
10 perpendicular axes x, y, and z. This movable base 29 provides another three degrees of 
freedom of movement to each PTZ system 20. Moreover, as shown in Fig. 14, instead of 
moving the camera and optics of a PTZ system 20, the PTZ system 20 could have a fixed 
camera 24 and fixed optics 22 and a movable mirror 21 positioned between them and the 
area to be monitored. In this embodiment, the movable mirror 21 provides the panning 
1 5 and tilting capability for each PTZ system 20. 

Fig. 2 provides a functional block diagram of a surveillance and monitoring 
system according to another preferred embodiment of the present invention. In the 
embodiment of Fig. 2, a WAIS 10 is used in combination with a single PTZ system 20. 
The WAIS 10 provides images to a display 50, preferably through an image processing 
20 unit 40. The display 50 is part of a monitoring station 80, which is attended to by an 
operator. The operator stationed at the monitoring station 80 observes the images 
provided by the WAIS 10, and when the operator detects a region of interest within the 
area being monitored and desires to get a better view of the region, the operator selects 
the region using input means 60. Input means 60 may include any well-know type of 
25 input device, such as a keyboard, a mouse, a joystick, or a touch-sensitive panel. The 
input means 60 communicates the input data provided by the operator to a coordinate 
mapping unit 70. Using the input data, the coordinate mapping unit 70 provides the 
appropriate pan, tilt, and zoom settings to the PTZ system 20, so that the PTZ system 20 
is directed to view the region of interest. The PTZ system 20 subsequently provides 
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high-resolution, zoomed images of the region of interest to the display 50. 

While it is possible to display the raw image output from the WAIS 10 on the 
display 50, preferably, as shown in Fig. 2, the images from the WAIS 10 are processed 
through an image processing unit 40 before they are displayed on the display 50. The 
5 image processing unit 40 maps the image data from the WAIS 10 into either a Cartesian- 
coordinate system to produce a perspective image or into a cylindrical-coordinate system 
to produce a panoramic image. Such mapping makes it easier for a human operator to 
interpret the image being displayed and, hence, to select a region of interest. 

The details of the perspective and panoramic mapping will depend on the exact 

1 0 type of WAIS 1 0 used, but the general principles are well known in the art and are 
described, for example, in S.E. Chen, "Quicktime VR - An Image-Based Approach to 
Virtual Environment Navigation", Proc. of SIGGRAPH 95, (8):29-38, August 1995. 
The general principles are also briefly illustrated in Figs. 3 A and 3B. Fig. 3A illustrates a 
perspective mapping of a scene S from a viewpoint V. The perspective mapping is the 

1 5 projection A onto a plane P, the projection A consisting of the points of intersection of 
plane P with rays from scene S passing through the viewpoint V. Fig. 3B illustrates a 
panoramic mapping of a scene S from a viewpoint V. The panoramic mapping is the 
projection A of the scene S onto a cylinder C surrounding the viewpoint V. The 
projection A consists of the points of intersection of the cylinder C with the rays from 

20 scene S passing through the viewpoint V. Advantageously, by mapping onto a cylinder, a 
complete 360 degree view of the scene may be obtained. The cylindrical projection may 
then be further mapped onto a plane for a panoramic display on a flat screen. The 
cylinder-to-plane mapping may be visualized by cutting the cylinder in Fig. 3B length- 
wise and flattening it out. 

25 To provide appropriate pan, tilt, and zoom settings to the PTZ system 20, the 

coordinate mapping unit 70 must map the coordinates of a region or object of interest in 
the WAIS 10 to the corresponding coordinates in the PTZ system 20. To avoid or 
simplify such mapping, it is preferred that PTZ system 20 should be placed very close to 
the WAIS 10. The close proximity of the PTZ system 20 and the WAIS 10 ensures that 
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the viewing directions of both systems are about the same. Therefore, mapping of object 
coordinates from the WAIS 10 to the PTZ system 20 involves little, or no, computation. 
Fig. 4 shows an especially preferred embodiment of an arrangement of a WAIS 10 and a 
PTZ system 20, in which the PTZ system 20 is placed directly underneath the WAIS 10. 
5 In practice, of course, it may be necessary to have one or more PTZ systems 

distributed around, instead of in close proximity to, a WAIS. In that case, to translate 
object coordinates between systems, assumptions must be made about the distance of 
objects from the WAIS 10 and the PTZ systems 20, given the geometry of the area being 
monitored. For example, assuming the area being monitored contains a flat, level floor, 

10 an assumption can be made that the objects of interest will be at or near the known floor 
level of the area being monitored (as in the case of humans walking about the area). 

Figs. 5 A and 5B illustrate the relationships between the coordinate systems of a 
WAIS 10 and a PTZ system 20 when the PTZ system 20 is not located near the WAIS 10. 
In the figures, an object 6 is observed in the WAIS 10 having coordinates (0 h (J),) and 

15 corresponding coordinates in the PTZ system 20 of (6 2 , (J> 2 ). The angles 0, and 0 2 define 
the coordinate angle of the object in the x-z plane, which is perpendicular to the ceiling 
and floor, and the angles 4>, and <J> 2 define the coordinate angle of the object in the x-y 
plane, which is parallel to the ceiling and floor. An assumption is made that the object 6 
is located at floor level (or at a fixed height from the floor level), which is a known 

20 perpendicular distance h^ from the WAIS 10 and a known perpendicular distance h pt2 
from the PTZ system 20. Using this assumption, the distance between the WAIS 10 and 
the object 6 along the x-axis, d wox , can be derived as shown in equation (1). 

d =h tan0, (1) 

wox wats 1 v 7 



Similarly, the distance between the PTZ system 20 and the object 6 along the x- 
axis, dpox, can be derived as shown in equation (2). 



d -h tan0 n 

pox ptz 2 



(2) 
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Using the relationship of equation (3), 

d = d + d (3) 

pox wox wpx x ' 

where is the known distance along the x-axis of the WAIS 10 from the PTZ system 
20, angle 0 2 can be derived by those of ordinary skill in the art as shown in equation (4). 

h . tanG, +d 

tanG (4) 

2 h v 7 

ptz 

5 In a similar manner, the angle <J> 2 can be derived as shown in equations (5), (6), 

and (7). 

d h tanG, 

, wox _ wats 1 , —v 

a r T yp) 

w °y tan^j tanc^ 



d = d ~ d 
poy woy wpy 



(6) 



d +d d +d h tanGtand) +</ tancj), 

i _ wox wpx _ wox wpx _ wats I * 1 wpx 1 ^— x 

an<P 2 ^ ~2 A tan9, -rf tan ({), 

poy woy wpy waif 1 wpy 1 



In equations (6) and (7), d^ is the known distance between the WAIS 10 and the 
PTZ system 20 along the y-axis. Using equations (4) and (7), therefore, the PTZ 
10 mapping unit 70 may map object coordinates from the coordinate system of the WAIS 10 
to that of the PTZ system 20. 

Fig. 6 provides a functional block diagram of a surveillance and monitoring 
system according to another preferred embodiment of the present invention. In this 
embodiment, the WAIS 10 provides image data to a frame grabber 30, which captures 
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image frames from the WAIS 10 at predetermined intervals. The frame grabber 30 
provides the image frames to a motion detection unit 92, which algorithmically detects 
the movement of objects within a series of image frames. The motion detection unit 92 
communicates with an object tracking unit 94, which tracks the detected objects. The 
5 object tracking unit 94 communicates with a coordinate mapping unit 95, which maps the 
coordinates of objects from the coordinate system of the WAIS 10 to that of the PTZ 
system 20, as discussed previously. 

When a single PTZ system is used to track multiple objects, as in the embodiment 
of Fig. 6, the PTZ system 20 must be time-shared among the objects being tracked. 
10 Accordingly, it is preferred that a PTZ scheduling unit 96 is included, which prioritizes 
the objects being tracked by the object tracking unit 94. The PTZ scheduling unit 96 
continuously updates the priority of objects being tracked based on information provided 
by the PTZ system 20 and the object tracking unit 94, such as the current position, 
velocity, and acceleration of the PTZ system 20 and the objects. 
1 5 The PTZ scheduling unit 96 communicates with a PTZ driver unit 98. Using the 

priorities set by the PTZ scheduling unit 96, the PTZ driver unit 98 sends appropriate 
commands to the PTZ controller of the PTZ system 20 so that the PTZ system 20 spends 
a predetermined amount of time on each object that is tracked. Alternatively, instead of 
using a single PTZ system and time-sharing among objects, multiple PTZ systems may 
20 be used to track multiple objects of interest simultaneously. 

As shown in Fig. 6, the output of the PTZ system 20 may be viewed on a display 
50 in real time. The display 50 may contain multiple windows for each of the objects 
being tracked. In addition, the output of the PTZ system 20 may also be recorded on 
recording equipment 52, such as a tape recorder or disk drive, for later viewing. If the 
25 output of the PTZ system 20 is recorded, advantageously a time stamp may also be 

recorded with the images. In addition, the output of the WAIS 10 may also be recorded 
for later viewing. By recording the output of the WAIS 10, a user is able to view not just 
detailed images of the objects of interest, but also their surroundings. 

In an exemplary embodiment of Fig. 6, the WAIS 10 comprises a 
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PARACAMERA from CYCLO VISION TECHNOLOGIES of New York, New York, 
used in conjunction with a Model GP KR222 camera from PANASONIC. The PTZ 
system 20 is a SONY CCD-IRIS camera mounted on a Model No. PTU-46-17.5 PTZ 
controller from DIRECTED PERCEPTION of Burlingame, California. The motion 
5 detection unit 92, object tracking unit 94, coordinate mapping unit 95, PTZ scheduling 
unit 96, and PTZ driver unit 98 are implemented in software within a general purpose 
computer 90. The general purpose computer 90 may be, for example, an INTEL 
PENTIUM PRO 200 MHZ system, running the MICROSOFT WINDOWS NT 4.0 
operating system, and including a MATROX PPB frame grabber add-on board from 

10 MATROX ELECTRONIC SYSTEMS LTD., 1055 St. Regis Blvd., Dorval, Quebec, 
Canada H9P 2T4. The display 50 and recording equipment 52 may also be part of the 
general purpose computer 90. 

Referring to Figs. 7 A through 7C, a flow-chart is shown detailing the steps of a 
preferred method for monitoring objects in an area. These steps may be programmed in 

15 software in a general purpose computer 90. An exemplary correspondence of the steps 
with the embodiment of Fig. 6 is shown by the dotted lines. Of course, it is clear that 
those skilled in the art may also easily implement the functionality of one or more of 
these steps in special-purpose hardware. 

In step 702, an image frame I t is retrieved from the frame grabber 30. If the image 

20 is a color image, the image frame I t may consist of separate color channels as, for 

example, red, green, and blue channels. In that case, all of the following computations 
may be performed with respect to each color channel. Alternatively, the color channels 
may be summed up to obtain a black and white image. 

Because images are prone to camera noise (i.e., changes in pixel intensities over 

25 time), in step 704, the image frame I t is filtered to remove any such noise. Any known 
noise filter may be used as. An exemplary filter is a box filter with a four-by-four (4x4) 
box size. This filter takes a four-by-four set of pixels in the image frame I t and generates 
a single output pixel that is the average of the intensities of the pixels in the set. Of 
course, this filter reduces the resolution of the image frame I t by four. The noise filter 
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produces a filtered image frame FI t . 

Since speed is critical in real-time applications, filtering is performed only in the 
image frame area that corresponds to the active imaging area of the WAIS 10. For 
example, if the WAIS 10 produces a circular image on a CCD, only the corresponding 

5 circular image area in the image frame I, is filtered. In addition, to further maximize 
performance, running balances may be maintained as the scan lines of image frame l t are 
read. For example, assuming that a four-by-four (4x4) box filter is used and that 
hypothetically the image frame I t is twelve (12) pixels wide, three variables s„ s 2 , and s 3 
may be used to store the sum of pixels 1 to 4, pixels 5 to 8, and pixels 9 to 12, 

10 respectively, when the first scan line of I t is read. When the second scan line of I t is read, 
the sums of pixels 1 to 4, pixels 5 to 8, and pixels 9 to 12 of the second scan line are 
added to s,, s 2 , and s 3 , respectively. Variables s,, s 2 , and s 3 are updated in the same 
manner when the third and fourth scan lines are read. After the fourth scan line is read, 
s,, s 2 and s 3 are divided by sixteen to obtain the box filter output for the first three four- 

1 5 by-four (4x4) boxes in the image frame I t . The process is repeated until all of the scan 
lines of I t are read. In this manner, memory access time is minimized. 

In step 706, the filtered image frame FI t is subtracted from the moving average 
image frame M t _, to produce a difference image frame D t . The moving average image 
frame M M represents a historical average of all past image frames. Thus, if the difference 

20 image frame D t is not zero, then something in the current image frame is different than in 
the past, and there is an indication of movement in the image. The reason the current 
image frame FI t is compared to the moving average image frame M M , instead of simply 
to the last image frame FI M , is to account for gradual changes in the lighting of the 
monitored area. 

25 In step 708, the moving average image frame is updated. Any moving average 

function may be used. An exemplary function is shown in equation (8). 

M - aM^ + (1 - a)FI t (8) 
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An exemplary value of a in equation (8) is 0.75. Using a ratio for a with a 
denominator that is an exponential value of two (2) is advantageous because it allows the 
use of binary shifting to perform multiplication and division by a, instead of using 
floating point operations. Binary shifting is the shifting of bits of a binary-represented 
5 number. As is well known by those in the art, each time a binary number is shifted to the 
left, the binary number doubles; and each time a binary number is shifted to the right, the 
binary number is divided by two. It is also well known that the use of binary shifting is 
faster than the use of floating point operations and, thus, minimizes calculation time. 

To account for any noise that may not have been filtered out in step 704, in step 
10 710, each pixel of the difference image frame D t is compared to a threshold value. Based 
on this comparison, a threshold image frame T t is created. Each pixel in T t has a value of 
"1" if the corresponding pixel in D t is above the threshold value and a value of "0" if the 
corresponding pixel in D t is below the threshold value. The pixels in T t with a value of 
"1" indicate motion in the corresponding region of I t . Advantageously, the step of 
1 5 comparing each pixel in D t to a threshold value and generating T t may be performed 
simultaneously with the step of subtracting FI t from M M . 

Once regions of motion are detected in T t , the regions must be associated with 
objects. This association is not as easy as associating each continuous region with an 
object because a single object may produce multiple regions of motion. For example, 
20 moving persons may produce movement in the regions of both their arms and legs. To 
determine which regions constitute a single object, image dilation is performed on T, in 
step 712. Image dilation consists of spreading or smearing those pixels in T t that indicate 
motion (i.e., have a value of "1")- For example, for each pixel indicating motion in T t , a 
box of pixels surrounding the motion-indicating pixel is selected, and the box is filled 
25 with the value "1". As a result of this image dilation process, separate regions merge 
together. The remaining continuous regions are characterized as single objects and are 
labeled for identification. 

Once objects are associated with regions of motion in T„ it is determined in step 
714 whether any of these objects correspond to objects that have been previously 
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identified. In general, such a determination will involve both temporal and spatial 
reasoning. A simple manner in which this determination can be made, however, is to 
compare the currently generated dilation frame with a previously-generated dilation 
frame and to assume that objects that overlap in the two frames are the same. Of course, 
5 more complicated and robust methods could also be used. For example, models of the 
objects could be built and could be tested against the behavior of objects in the current 
and past frames. After the current objects are correlated with the past objects, all unique 
objects are labeled for identification. 

In step 716, a selection is made as to which objects to track. The selection criteria 
10 is necessarily application specific, but examples of such criteria are the size, shape, 
duration of existence, and location of objects. 

In step 71 8, specific viewing parameters are generated for each object to be 
tracked. That is, each object is usually spread out over some area in the image. In this 
step, it is determined on which point of the object to focus. Such a determination may be 
1 5 performed, for example, by determining the centroid (center of mass) of each object's 
area in the dilated image frame. Alternatively, the center of the bounding box for each 
object in the dilated image frame may be used. More preferably, a weighted average of 
the centroid and the center of the bounding box may be used. 

It is possible that, because of the time required for the calculations described 
20 above, the viewing parameters determined in step 718 will not generate a smooth 

trajectory for an object over time. Therefore, the video image of an object as viewed 
from the PTZ system may appear "jerky." To avoid this jerkiness, in step 720, a 
smoothing function is applied to the viewing parameters generated in step 718. Kalman 
filters, such as Wiener filters, have been tried and have been found to work robustly. A 
25 disadvantage to using these smoothing filters is that the smoothed viewing position of the 
object may lag behind the actual position of the object. To compensate for this effect, a 
feed-forward term is preferably added to the output of these smoothing filters. An 
exemplary feed-forward term is twice the difference between the previous smoothed 
viewing position and the previous actual object position. In addition, an adaptive filter, 
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which changes the rate of acceleration of the viewing position based on the rate of 
acceleration of the object may also be used. 

In step 722, optionally, a perspective or panoramic display of the objects being 
tracked may be generated from the image frame I t provided by the wide-angle imaging 
5 system based on the viewing parameters generated in step 720 and the size and center of 
the objects being tracked. Perspective and panoramic mapping is performed as discussed 
in relation to Figs. 3A and 3B. Preferably, each object is displayed in its own window. 

In step 724, the smoothed viewing parameters are mapped from the WAIS 
coordinate system to the PTZ coordinate system, as described previously. In step 726, 
10 scheduling of the objects is performed. A simple first-in, first-out (FIFO) queuing system 
in which the first-observed objects receive the highest priority may be used. Of course, 
more complicated heuristics could also be implemented. Once scheduling is completed, 
in step 728, the PTZ driver unit 98 converts the PTZ object coordinates into 
manufacturer-specific codes for the PTZ controller of the PTZ system 20. 
1 5 Fig. 8 illustrates a preferred embodiment of a wide-angle imaging system for use 

with the present invention. The WAIS 10 comprises a camera 100 having a telecentric 
means 1 1 0 attached to it, which are positioned below a paraboloid-shaped mirror 120. As 
described fully in U.S. Patent Application No. 08/644,903, filed on May 10, 1996, 
entitled "An Omnidirectional Imaging Apparatus," and in the continuation-in-part 
20 application of the same title, U.S. Patent Application No. 08/986,082, filed on December 
5, 1997, both of which are incorporated by reference herein, the paraboloid-shaped mirror 
120 orthographically reflects any principal rays of a scene that would otherwise pass 
through the focus of the paraboloid. The telecentric means 110 filters out any rays of 
light that are not orthographically reflected by the paraboloid-shaped mirror 120. The 
25 telecentric means 1 10 may be a telecentric lens, a telecentric aperture, or a collimating 
lens used in combination with imaging optics. 

When the paraboloid-shaped mirror is truncated at a plane passing through its 
focus and normal to its paraboloidal axis, the paraboloid-shaped mirror is capable of 
orthographically reflecting principal rays from an entire hemisphere. Advantageously, 
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therefore, the WAIS 10 using the paraboloid-shaped mirror 120 is able to provide a 
hemispherical field of view from a single viewpoint (i.e., from the focus of the 
paraboloid). Moreover, since the WAIS 10 has a single viewpoint, its images can be 
mapped to perspective and panoramic views without any distortion. 
5 Alternatively, other catadioptric imaging systems may be used in the present 

invention. As used in this specification and the appended claims, the term "catadioptric" 
refers to an imaging system that uses a combination of reflecting surfaces (such as 
mirrors) and refracting surfaces (such as lenses). Examples of other catadioptric imaging 
systems that may be used as the wide-angle imaging system of the present invention 
10 include, as shown in Fig. 9, a hyperboloidal or ellipsoidal mirror 140 used in conjunction 
with a perspective lens 130. Examples of hyperboloidal and ellipsoidal mirrors are 
disclosed in Yamazawa et al. and Rees, which have been discussed earlier. 
Hyperboloidal and ellipsoidal systems may also be configured to provide a single 
viewpoint and, thus, as with paraboloidal systems, the images produced by such systems 
1 5 may be mapped to perspective or panoramic views without distortion. Nonetheless, these 
systems are less favored than paraboloidal systems because they require complex 
calibration and implementation. 

Additionally, although it is less desirable, catadioptric systems containing other 
curved surfaces that do not provide images from a single viewpoint, such as spherical or 
20 conical surfaces, may also be used for the wide-angle imaging system of the present 
invention. Although they do not provide images with a single viewpoint, such systems 
are capable of serving the main purpose of the wide-angle imaging system of the present 
invention~i.e., to detect regions of interest (and not necessarily to provide distortion-free 
visual details of those regions). Thus, although it may be desirable, it is not critical to 
25 provide images with a single viewpoint from the wide-angle imaging system of the 
present invention. 

In addition to curved surfaces, various arrangements of planar mirrors may be 
used for the wide-angle imaging system of the present invention. For example, in Fig. 
10A, triangular planar mirrors 150a are arranged in a pyramid shape to reflect a wide- 
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angle field of view to a single camera 100. As another example of a catadioptric system 
with planar mirrors, in Fig. 10B, polygonal planar mirrors 150b are arranged in a 
polyhedral shape to reflect a wide-angle field of view to multiple cameras 100. A 
specific example of this embodiment is disclosed in U.S. Patent No. 5,539,483 to Nalwa, 

5 which was discussed earlier. 

Instead of a catadioptric system, commercial wide-angle lenses may also be used 
for the wide-angle imaging system of the present invention. For example, as shown in 
Fig. 1 1, a wide-angle lens 160, such as a 2mm focal-length COMPUTAR EMH200, 
could be mounted on a camera 100 on a ceiling 5 to provide a 1 15° view of the area 

10 beneath it. Alternatively, if a wider field of view is necessary, a fish-eye lens 170, as 
illustrated in Fig. 12, such as a NIKON 8mm focal-length F2.8 lens, could be used to 
provide a substantially hemispherical field of view. 

In addition, as shown in Figs. 15A and 15B, the wide-angle imaging system of the 
present invention may comprise multiple camera systems 200. The camera systems 200 

1 5 may contain non-overlapping fields of view as in Fig. 1 5 A, or they may contain 

overlapping fields of view as in Fig. 15B. In addition, each camera system 200 may be 
either a narrow-angle or wide-angle camera system. 

Although the present invention has been described with reference to certain 
preferred embodiments, various modifications, alterations, and substitutions will be 

20 known or obvious to those skilled in the art without departing from the spirit and scope of 
the invention, as defined by the appended claims. 
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CLAIMS 

1 . A surveillance and monitoring system for monitoring an area, comprising: 
a first imaging system having a wide-angle field of view approximately 

equal to or greater than said area; 

one or more second imaging systems having adjustable view settings, each 
of said one or more second imaging systems positioned to view portions of said area and 
being capable of producing images of said portions with a resolution greater than said 
first imaging system; and 

one or more controls for controlling said adjustable view settings of said 
one or more second imaging systems. 

2. The surveillance and monitoring system of claim 1 , wherein said area is 
substantially hemispherical and said first imaging system has a substantially 
hemispherical field of view. 

3. The surveillance and monitoring system of claim 1, wherein said 

15 adjustable view settings include a pan setting for adjusting the pan angle of said one or 
more second imaging systems with respect to said area. 

4. The surveillance and monitoring system of claim 1, wherein said 
adjustable view settings include a tilt setting for adjusting the tilt angle of said one or 
more second imaging systems with respect to said area. 

20 5. The surveillance and monitoring system of claim 1 , wherein said 

adjustable view settings include zoom settings for adjusting the zoom of said one or more 
second imaging systems with respect to said portions of said area. 

6. The surveillance and monitoring system of claim 1 , wherein said first 
imaging system is a catadioptric system comprising a reflecting surface positioned to 
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reflect an image of said area and one or more image sensors optically coupled to said 
reflecting surface. 

7. The surveillance and monitoring system of claim 6, wherein said reflecting 
surface comprises a truncated, substantially paraboloid-shaped mirror. 

5 8. The surveillance and monitoring system of claim 6, wherein said reflecting 

surface comprises a truncated, substantially hyperbololdal-shaped mirror. 

9. The surveillance and monitoring system of claim 6, wherein said reflecting 
surface comprises a substantially ellipsoidal-shaped mirror. 

10. The surveillance and monitoring system of claim 6, wherein said reflecting 
10 surface comprises one or more curved mirrors. 

1 1 . The surveillance and monitoring system of claim 6, wherein said reflecting 
surface comprises one or more planar mirrors. 

12. The surveillance and monitoring system of claim 1 , wherein said first 
imaging system comprises imaging optics and one or more image sensors optically 

1 5 coupled to said imaging optics. 

13. The surveillance and monitoring system of claim 12, wherein said imaging 
optics comprises a wide-angle lens. 

14. The surveillance and monitoring system of claim 12, wherein said imaging 
optics comprises a fish-eye lens. 
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15. The surveillance and monitoring system of claim 1 , wherein said first 
imaging system comprises a plurality of camera systems. 

16. The surveillance and monitoring system of claim 1, further comprising a 
monitoring station including: 

5 one or more displays, at least one display coupled to said first imaging 

system for displaying an image of said area and at least one display coupled to said one or 

more second imaging systems; and 

input means for selecting a region of interest displayed on said at least one 

display coupled to said first imaging system, said input means operatively coupled to said 
1 0 one or more controls for controlling said adjustable view settings such that at least one of 

said one or more second imaging systems is positioned to view said region of interest. 

17. The surveillance and monitoring system of claim 1 , wherein said first 
imaging system provides an image signal representative of said area, and further 
comprising an image signal processing apparatus coupled to said first imaging system to 

1 5 convert said image signal into image signal data. 

1 8. The surveillance and monitoring system of claim 17, wherein said image 
signal processing apparatus includes means for mapping said image signal data into a 
Cartesian-coordinate system to produce a perspective image. 

19. The surveillance and monitoring system of claim 17, wherein said 

20 image signal processing apparatus includes means for mapping said image signal data 
into a cylindrical-coordinate system to produce a panoramic image. 

20. The surveillance and monitoring system of claim 1, further comprising 
motion detection means coupled to said first imaging system for detecting objects in 
motion in said area. 
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2 1 . The surveillance and monitoring system of claim 20, further comprising 
tracking means for tracking one or more of said objects in motion in said area, said 
tracking means having an input coupled to said motion detection means and an output 
coupled to said one or more controls for controlling said adjustable view settings. 

5 22. A surveillance and monitoring system for monitoring an area, comprising: 

an imaging system having a wide-angle field of view approximately equal 

to or greater than said area; 

motion detection means coupled to said imaging system for detecting 

objects in motion in said area; 
1 o tracking means coupled to said motion detection means for tracking one or 

more of said detected objects in motion; and 

image processing means coupled to said imaging system and said tracking 

means for producing perspective images of said tracked objects from images provided by 

said imaging system. 

23. A method for monitoring an area, comprising the steps of: 

(a) sensing a global image of said area with a first imaging system 
having a field of view approximately equal to or greater than said area; 

(b) detecting a region of interest within said global image; 

(c) sensing one or more detailed images of said region of interest with 
at least one of said one or more second imaging systems, at least one of said detailed 
images having a higher resolution than said global image. 

24. A method for monitoring an area, comprising the steps of: 
(a) positioning a first imaging system to view said area, said first 

imaging system having a wide-angle field of view approximately equal to or greater than 
25 said area; 
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(b) positioning one or more second imaging systems having adjustable 
view settings to view portions of said area, each of said one or more second imaging 
systems being capable of producing images of said portions with a resolution greater than 
said first imaging system; 
5 (c) sensing an image of said area with said first imaging system; 

(d) detecting a region of interest within said sensed image; and 

(e) controlling said adjustable view settings so as to view said region 
of interest with at least one of said one or more second imaging systems. 

25. A method for monitoring one or more objects in motion in an area, 
1 0 comprising the steps of: 

(a) positioning a first imaging system to view said area, said first 
imaging system having a wide-angle field of view approximately equal to or greater than 
said area; 

(b) positioning one or more second imaging systems having adjustable 
15 view settings to view portions of said area, each of said one or more second imaging 

systems being capable of producing images of said portions with a resolution greater than 
said first imaging system; 

(c) sensing images of said area with said first imaging system; 

(d) detecting said one or more objects in motion from said sensed 

20 images; 

(e) tracking one or more of said detected objects; and 

(f) controlling said adjustable view settings so as to view said tracked 
objects with at least one of said one or more second imaging systems. 

26. The method for monitoring one or more objects in motion in an area of 
25 claim 25, wherein said step of detecting comprises: 

generating a series of image frames at predetermined time intervals from 
said sensed images; 
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filtering said series of image frames for noise; 

calculating a moving average frame from a subset of said series of image 
frames comprising the set from the first image frame to the next-to-last image frame; 

subtracting the filtered, last image frame of said series of image frames 
5 from said moving average frame to produce a.difference image frame; 

comparing each pixel of said difference image frame to a predetermined 
threshold value to produce a threshold image frame indicative of regions of motion in 
said area; 

defining a first group of objects associated with said subset of said series 

10 of image frames; 

associating a second group of objects with said regions of motion in said 

threshold image frame; and 

generating a third group of objects comprising all unique objects in said 

first and second groups. 

15 

27. The method for monitoring one or more objects in motion in an area of 
claim 26, wherein the step of tracking comprises: 

selecting a fourth group of objects to be tracked from said third group of 
objects on the basis of a predetermined criteria; 
20 determining a focus point for each object in said fourth group of objects; 

and 

applying a smoothing function to the focus point of each object. 

28. The method for monitoring one or more objects in motion in an area of 
claim 25, wherein said step of controlling comprises mapping the coordinates of said 

25 tracked objects from the coordinate system of said first imaging system into the 
coordinate system of at least one of said one or more second imaging systems. 
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